fix(installer): resolve race condition and fs detection failure when mounting ESP - #111

Pushed 6d8ac0f to this branch: the ESP mount at configurator:775 now says mount -t vfat.

The description's "After" block already has that line, but the diff did not — it added udevadm settle and sleep 2 and left the mount untyped. Worth being explicit about why that matters, because the two changes fix different things and only one of them fixes this bug.

Can't find a SQUASHFS superblock is not a claim about what is on the partition. It is what an untyped mount says at the end of a guess: when libblkid hands back no type, or an ambiguous one, mount(8) walks the non-nodev types in /proc/filesystems trying each, and on the live ISO squashfs is in that list because the airootfs is a squashfs image. So the error names the last thing tried, not the thing that failed. Waiting does not change what the probe sees — and the reports this matches (omacom/omarchy#7263, omacom/omarchy#7515) are both deterministic, reproducible on every attempt until the partition layout changed, which a timing race would not be. Both are disks where the free space was carved out of an existing layout, which is exactly the case that leaves libblkid with more than one signature to choose between.

mkfs.fat -F32 runs three lines above, so there is nothing to guess at. Naming the type skips the probe entirely, and a filesystem that genuinely is not readable now fails as one instead of as a squashfs that was never there.

Two things left for you, neither of which I changed:

• sleep 2 at line 774 is two seconds added to every install, on every machine, after udevadm settle has already blocked until the udev queue drained. If settle was not enough, a fixed timer is not a fix either — it is a guess at how long "not enough" lasts. I would drop it, but it is your call and it is harmless enough that I did not touch it.
• udevadm settle at line 773 is unguarded, unlike every other udev/partprobe call in this file (lines 548, 697, 909, and wait_for_device in disk-partitioning.sh, all 2>/dev/null || true). There is no set -e here so a failure is ignored anyway, but its stderr goes straight to the tty gum is drawing on rather than into /var/log/omarchy-install.log — that is the whole reason disk_step exists. udevadm settle 2>/dev/null || true would match the file. It can also block for up to its 120s default on an unrelated queued event.

The -t vfat in detect_windows_esp is fine and I left it alone. The loop already filters on blkid -t TYPE=vfat, so mount would have inferred vfat regardless and the flag changes no outcome on the success path; on the failure path it fails cleanly instead of falling through the same /proc/filesystems list. No regression risk for FAT16 or FAT12 ESPs either — the kernel's vfat driver covers them, and exfat is reported as exfat by blkid so it never enters the loop.

Tests: ./test/all passes on a clean VM before and after the pushed commit (58 python tests, plus cidata-load and partition-numbering). None of them reach this code path — ./test/integration would, but it needs a built ISO and QEMU.

Thanks for the detailed feedback! I've resolved the merge conflict.

Re-reviewed at daa0cc3. The delta is the removal of udevadm settle and sleep 2, and it keeps 6d8ac0f — I checked that 6d8ac0f is still an ancestor of daa0cc3 with its tree intact, that the branch was not rewritten, and that line 775 still reads mount -t vfat "$efi_dev". Nothing else moved. Against quattro the PR is now +4/-2 in one file: -t vfat on the two mounts, plus the comment above the ESP one. Both items I left for you are closed, and the removal cannot regress anything relative to the merge target, since neither line exists on quattro in the first place.

On the race the title names, for the record: there was no window to guard at 775 even before you removed those lines. create_partition already runs partprobe and udevadm settle right after parted (disk-partitioning.sh:93-95); configurator:718-720 runs partprobe, sync and a sleep 2; wait_for_device at 722 then polls [[ -b ... ]] for ten seconds with another settle between tries and aborts the install if the node never appears. After that, wipefs -af at 727 and mkfs.fat -F32 at 770 each open that device read-write and each abort through disk_step on failure. By the time the mount runs, the node has been opened and written successfully twice, and mkfs.fat fsyncs before it exits. A fourth settle had nothing left to learn.

Naming the type is what changes the outcome, and it does not merely make the probe faster — it removes it. With -t vfat, mount(8) hands the type straight to mount(2) and never probes the source with libblkid, so a probe coming back empty or ambiguous on a slow-enumerating USB or NVMe cannot reach this call site at all. It also gives the kernel a name to autoload: get_fs_type() calls request_module("fs-vfat") for an unregistered driver, which the untyped path cannot trigger, because the fallback only tries filesystems already listed in /proc/filesystems. On the live ISO squashfs is in that list — the airootfs is a squashfs image (profiledef.sh:14) — while vfat generally is not, since nothing mounts a FAT volume during a normal live boot.

One thing I could not establish, and would rather say than leave implied. For the untyped mount to fail, that libblkid probe has to come back empty or ambiguous, and I could not reproduce that condition. On util-linux 2.42 I built a loop device carrying a real btrfs filesystem and ran mkfs.fat -F32 over it without wiping — the "free space carved out of an existing layout" case the reports describe — and mkfs.fat erased the competing signature itself; blkid returned an unambiguous vfat and the untyped mount succeeded. Planting a foreign magic by hand did not fool the prober either. So the fix is right and strictly more deterministic than probing, but the original trigger is still not pinned down, and wipefs -af "$efi_dev" at line 727 (added in 7d3b01e, before this PR) may already cover part of it.

detect_windows_esp still enumerates with blkid -t TYPE=vfat -o device at 393, which genuinely can come back empty on a device that has not enumerated yet — harmless, because its only consumer is the say line at 539 and Omarchy never adopts an existing ESP (comment at 531-536).

What ran: ./test/all on a clean disposable VM at daa0cc3 — shell unit tests plus 58 python tests, all green — and bash -n on the configurator. None of it reaches the changed code. That path is the installer's disk stage and needs a booted ISO; ./test/integration would cover it but wants a built ISO and QEMU, so it did not run. What I could execute is the mount call itself: on a loop device, mkfs.fat -F32 followed immediately by mount -t vfat, no settle and no sleep, succeeds. That is evidence the removal is safe at the level of the mount, not evidence the install works end to end.

Second opinion from codex at xhigh: no defects, and it agreed with the conclusions above — though its independence is not currently guaranteed, as it can read this session's own transcript. Three things it contributed that I had not reasoned about, all verified against the source: the settle inside create_partition, above; that a mount failure at 775 is not merely an abort but a rollback, since disk_step calls disk_abort_hook at 426, which unmounts /mnt, closes LUKS and reclaims the partitions this run created, so a wrong -t backs out cleanly rather than leaving a half-built disk; and that the live ISO boots linux-t2 rather than stock linux (builder/build-iso.sh:121), which is where vfat's availability actually rests — a module set this checkout does not vendor and that can only be confirmed against a built ISO. Not a dependency this PR introduces. I did not take its citations: the line ranges it gave for the kernel and dosfstools sources do not exist in those files, even though the claims themselves hold.

Left to do on your side: the PR closes omacom/omarchy#7263 and omacom/omarchy#7515 and references neither, so neither reporter will find it. A Fixes line for both is the last thing outstanding. Merging is the maintainer's call, not mine.

I was able to edit my configurator to match this PR from root after hitting this bug and get through install. Thanks, @Damian626

Confirming this fix works. Hit the identical failure on omarchy-4.0.0.iso (built 2026-08-14): free-space install to an external USB SSD (GPT, three existing NTFS partitions, new partitions landing in slots 3/4), encrypted mode. Failed reproducibly 4+ times with mount: /mnt/boot: fsconfig() failed: Can't find a SQUASHFS superblock on sda3 LUKS/btrfs steps all succeeded, and dd-verified the FAT boot sector was correctly written at the partition start, so it's purely a detection failure. Live-patched the configurator to mount -t vfat (plus sync; udevadm settle) and the install completed first try and boots fine.

Re-reviewed at daa0cc3. No code has moved since the last pass, so this is only about what arrived around it.

The Fixes basecamp/omarchy#7263 and Fixes basecamp/omarchy#7515 lines are in the body now — that was the last thing outstanding on your side, and it is done. One caveat that belongs to whoever merges rather than to you: those issues are in a different repository, and GitHub documents the KEYWORD OWNER/REPO#NUMBER syntax but does not state that the automatic close on merge into the default branch extends across repositories. Both issues are still open, so they are worth checking after the merge and closing by hand if they stay open.

Two confirmations arrived since (neilcode, ShivanshTiwari613). I checked them against this head rather than taking them at face value, and the detailed one is consistent in every particular the source can check: /mnt/boot is the ESP target only in encrypted mode (configurator:660), the ESP is created before the root partition and so takes the lower of the two new GPT slots — sda3 with root at sda4, since parted fills the lowest free slot (disk-partitioning.sh:3-7) — and LUKS at :731 and btrfs at :749 both complete before the ESP is formatted at :772 and mounted at :775, so "LUKS and btrfs succeeded, /mnt/boot failed" is exactly the order this file runs in. The dd-verified boot sector is the part that carries weight: it puts the failure in detection rather than in the write, which is what naming the type addresses and what a settle or a sleep could not.

Those reports also close two things the last review had to leave open. The live ISO boots linux-t2 rather than stock linux (builder/build-iso.sh:121, and it has been the ISO kernel since 2025-09-14), and whether vfat is available under that kernel's module set was not confirmable from this checkout; a live-patched mount -t vfat completing an install on the 4.0.0 ISO answers it from the field. It also narrows the root cause. wipefs -af "$efi_dev" at :728 arrived in 7d3b01e on 2026-08-13, and before that commit only the root device was wiped, never the ESP — so if the 2026-08-14 ISO was cut after it, the ESP was wiped, FAT32 was written, and the untyped mount still fell through, which would rule out a leftover signature as the whole story. A build date alone does not pin the commit, so that stays open.

One caveat on the confirmation itself: ShivanshTiwari613's live patch bundled sync; udevadm settle with the typed mount, and this branch has neither, since daa0cc3 removed them. So that report is not a test of the branch as it stands. It does not need to be — with -t vfat, mount(8) hands the type to mount(2) and never probes the source, mkfs.fat fsyncs before it exits, and create_partition plus :718-722 had already settled and waited on that node before the format. But it is a bundled change, and reading it as proof that these four lines are sufficient would be reading past the confound.

Nothing was executed this run: the head is unchanged, so the ./test/all green at daa0cc3 from the previous pass is what stands, and the changed path is still the installer's disk stage, which needs a booted ISO. Two independent field reports are the closest thing to an end-to-end result this change has, and they are reports rather than a run here.

Codex at xhigh reviewed both comments against the source as an independent second opinion. It found nothing that changes the verdict, agreed with the consistency checks above, and added one thing I had not: mkfs.fat is dosfstools 4.2-5 on this ISO (manifests/fresh-4.json) and fsyncs the device before returning, so the sync in that live patch adds no ordering the following mount needs. Its independence is not currently guaranteed. I did not take its reading of cross-repository auto-closing — the documentation it cited does not say what it said it says.

Merging remains the maintainer's call.

I was preparing to submit this same mount -t vfat change, but found this PR while checking for duplicates.

I independently reproduced and verified the fix on real hardware during a free-space Omarchy installation with two NVMe drives connected. The stock installer created the new 2 GiB FAT32 OMARCHY_EFI partition, but the subsequent untyped mount failed with the SQUASHFS superblock error.

I changed only the target ESP mount to use mount -t vfat, then repeated the installation with both NVMe drives still connected. The installation completed successfully and the system booted normally. The encrypted Btrfs root is working, and the new vfat ESP is mounted at /boot.

So this PR’s explicit vfat mount resolves the failure I encountered on real hardware. Thank you for putting the fix together.

Re-reviewed at daa0cc3. No code has moved: this is the same head as the last two passes, and the commit titled "resolved merge conflict and applied suggestions" is daa0cc3 rather than something newer. I re-checked both typed mounts out of the working tree anyway rather than trusting the earlier verdict — a conflict resolution is exactly where a fix gets silently dropped. Both survive it: configurator:396 is mount -t vfat -o ro "$p" "$tmp_mp" and configurator:775 is mount -t vfat "$efi_dev" /mnt"$esp_mount_in_target". 6d8ac0f is still an ancestor and the branch was not rewritten.

What is new is underneath you. quattro has advanced three commits (df80b32, 6d02625, 268bac1) since the merge base, none of which touches configurator. The merge is clean, and I ran the suite against the merged tree — not this branch in isolation — on a disposable VM: test/all green, 63 python tests (up from 58, because the base added its own). That is the tree that would actually land. It still does not reach the changed path, which is the installer's disk stage and needs a booted ISO; ./test/integration would cover it but wants a built ISO and QEMU.

On the race in the title. The diff adds no wait, no retry, no barrier — so it is worth being precise about what does establish the ordering, because a retry loop that wins on fast hardware is the usual near-miss here and this is not that. wait_for_device (disk-partitioning.sh:45-53) is a bounded 10x1s poll, but it is not the last word before the mount. Between it and :775, wipefs -af "$efi_dev" at :728 and mkfs.fat -F32 at :772 each open that block device and write to it, and each is wrapped in disk_step, which returns only on status zero and otherwise aborts through disk_abort_hook (disk-partitioning.sh:70-77, configurator:427-435). Reaching :775 at all therefore proves the node existed and was written successfully twice. That is a genuine happens-before from a completed mkfs.fat, not a timer that got lucky. The fix does not synchronize a race; it removes the guess, which is the half that was actually broken.

The mechanism, verbatim from mount(8) as shipped (util-linux 2.42.2; the manifest pins 2.42.1-1): "If no -t option is given, or if the auto type is specified, mount will try to guess the desired type. mount uses the libblkid(3) library for guessing the filesystem type; if that does not turn up anything that looks familiar, mount will try to read the file /etc/filesystems, or, if that does not exist, /proc/filesystems. All of the filesystem types listed there will be tried, except for those that are labeled nodev." On the live ISO squashfs is in that list. With -t vfat none of it runs.

@mateo-bolanos — thank you, and your report carries more weight than it may look like. Two reasons. First, every particular of it checks out against this source: the 2 GiB FAT32 OMARCHY_EFI partition is EFI_SIZE_B=$((2 * gib)) at :594, created at :704 and formatted at :772; and /boot is the ESP target only in encrypted mode (:660, unencrypted gets /efi at :662), which matches your encrypted Btrfs root. Your two-NVMe free-space setup is also omacom/omarchy#7263's configuration exactly — two NVMe drives, Windows on one, free space carved out on the other, LUKS and btrfs and the subvolumes all succeeding before it dies on the ESP mount. So this is a field reproduction of one of the two issues this PR closes.

Second, and this is the part that matters: you changed only the mount. ShivanshTiwari613's confirmation above bundled sync; udevadm settle with the typed mount, and this branch has neither — daa0cc3 removed them — so that report, useful as it was, is not a test of the branch as it stands. Yours is a single-variable test of precisely the line at :775. It is the first field confirmation that does not have a confound in it.

What it does not settle, and nothing yet has: why the untyped probe comes back empty or ambiguous in the first place. You reproduced the failure with both drives connected and fixed it with both connected, which does not isolate the second drive as the trigger. Two earlier attempts to reproduce the ambiguity on a loop device failed. The fix is right and is strictly more deterministic than probing either way, but the root cause stays unpinned.

On #113, since it is changing how the target is installed and touches ESP handling: it does not collide with this. Textually its only configurator hunk is at @@ -1083,6 +1083,23 @@, a medium-verify gate ahead of run_partition_execute, nowhere near :396 or :775. The interesting part is that it could have collided and does not. _install_root_image in orchestrator/phases_impl.py unmounts the whole target layout and replays it to swap the received image in as @ — a replay that reintroduced an untyped ESP mount would have undone this PR on the free-space path. It reads the table with findmnt -R -J -o TARGET,SOURCE,FSTYPE,OPTIONS and remounts each entry with mount -t <fstype>, and its own test asserts the ESP comes back as mount -t vfat ... /boot. The type it replays is whatever mounted the ESP originally, which on that path is vfat because of this PR. The two compose. (#122 and #117 also touch configurator, at :507/:896 and :870 — no overlap either.)

Reviewed by Claude Opus 5 and by Codex at xhigh as a second opinion. Codex found no defects in the six changed lines, and agreed with the ordering argument above — its independence is not currently guaranteed, since its read-only sandbox still let it read the triage protocol this review follows. One thing it added that I had not reasoned about: -t does not switch libblkid off wholesale, only superblock probing — libblkid is still used to resolve a UUID=/LABEL= source. It makes no difference here, because both call sites pass a device path, but "removes the probe entirely" was looser than the truth. I did not take its upstream context.c and vfat.c line citations; I verified the behaviour from the mount(8) page on the shipped util-linux instead, and a previous pass found comparable citations from it pointing at lines that do not exist.

Nothing pushed, nothing outstanding on your side. basecamp/omarchy#7263 and #7515 are both still open, and since they live in another repository they are worth checking by hand after the merge rather than assuming the Fixes lines close them. Merging is the maintainer's call.